On the performance of social network and likelihood-based expert weighting schemes

Using expert judgment data from the TU Delft's expert judgment data base, we compare the performance of different weighting schemes, namely equal weighting, performance based weighting from the classical model (Cooke, 1991), social network weighting and likelihood weighting. The picture that emerges with regard to social network weights is rather mixed. SN theory does not provide an alternative to performance based combination of expert judgments, since the statistical accuracy of the SN decision maker is sometimes unacceptably low. On the other hand, it does outperform equal weighting in the majority of cases. The results here, though not overwhelmingly positive, do nonetheless motivate further research into social interaction methods for nominating and weighting experts. Indeed, a full expert judgment study with performance measurement requires an investment in time and effort, with a view to securing external validation. If high confidence in a comparable level of validation can be obtained by less intensive methods, this would be very welcome, and would facilitate the application of structured expert judgment in situations where the resources for a full study are not available. Likelihood weights are just as resource intensive as performance based weights, and the evidence presented here suggests that they are inferior to performance based weights with regard to those scoring variables which are optimized in performance weights (calibration and information). Perhaps surprisingly, they are also inferior with regard to likelihood. Their use is further discouraged by the fact that they constitute a strongly improper scoring rule.